🕷️️ Job Radar • SCRAPING

freelancer.com 🟠 2026-05-09

🔹 Thorough Scrape and Front-end Teardown of Commercial Astrology Website
👤 Client: 🇮🇳 New Delhi, India Member since 2026-04-17
💰 Price: $26 Average bid
🚩 Problem: Capture comprehensive data on user interactions and platform design to understand engagement and service presentation.
📦 Existing: Not specified

Specifications:

[Target] Full site coverage, including horoscope generators, pricing tables, sign-up flows, interactive elements, and embedded content.
[Method] Use Python-Scrapy or Playwright for dynamic scraping; BeautifulSoup for static pages. Ensure rate limiting to avoid CAPTCHA triggers.
[UI/UX] Document design patterns, wording, sequence of steps, and interactive behaviors (e.g., birth date entry, paywall appearance).
[Stack] Python-Scrapy, BeautifulSoup, Playwright, SQLite or PostgreSQL for database storage, Pandas for data manipulation.
[Security] Respect rate limits to avoid CAPTCHA. Ensure no server-side probing.
[Format] CSV/JSON for structured data dump; PDF/DOCX for report; HTML archive for reference.

Workflow:

1. Define scope and set up scraping environment with Python-Scrapy or Playwright, ensuring compatibility with dynamic content.
2. Crawl the entire site, capturing all visible elements including horoscopes, pricing tables, sign-up flows, and interactive behaviors.
3. Extract text, image URLs, and link destinations using BeautifulSoup for static pages and Playwright for dynamic interactions.
4. Store data in a structured format (CSV/JSON) and organize by sections (home, user account, horoscope, shop).
5. Create a navigable site map with URL hierarchy and generate screenshots of key workflows such as account creation and checkout.
6. Write a detailed report (PDF/DOCX) illustrating each major feature and function with annotated screenshots.
7. Archive raw HTML files for reference.